

<!DOCTYPE html>
<html class="writer-html5" lang="en" >
<head>
  <meta charset="utf-8" />
  
  <meta name="viewport" content="width=device-width, initial-scale=1.0" />
  
  <title>监视器故障排除 &mdash; Ceph Documentation</title>
  

  
  <link rel="stylesheet" href="../../../_static/ceph.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/pygments.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/graphviz.css" type="text/css" />
  <link rel="stylesheet" href="../../../_static/css/custom.css" type="text/css" />

  
  
    <link rel="shortcut icon" href="../../../_static/favicon.ico"/>
  

  
  

  

  
  <!--[if lt IE 9]>
    <script src="../../../_static/js/html5shiv.min.js"></script>
  <![endif]-->
  
    
      <script type="text/javascript" id="documentation_options" data-url_root="../../../" src="../../../_static/documentation_options.js"></script>
        <script src="../../../_static/jquery.js"></script>
        <script src="../../../_static/underscore.js"></script>
        <script src="../../../_static/doctools.js"></script>
    
    <script type="text/javascript" src="../../../_static/js/theme.js"></script>

    
    <link rel="index" title="Index" href="../../../genindex/" />
    <link rel="search" title="Search" href="../../../search/" />
    <link rel="next" title="OSD 故障排除" href="../troubleshooting-osd/" />
    <link rel="prev" title="Ceph 社区" href="../community/" /> 
</head>

<body class="wy-body-for-nav">

   
  <header class="top-bar">
    

















<div role="navigation" aria-label="breadcrumbs navigation">

  <ul class="wy-breadcrumbs">
    
      <li><a href="../../../" class="icon icon-home"></a> &raquo;</li>
        
          <li><a href="../../">Ceph 存储集群</a> &raquo;</li>
        
          <li><a href="../../operations/">集群运维</a> &raquo;</li>
        
      <li>监视器故障排除</li>
    
    
      <li class="wy-breadcrumbs-aside">
        
          
            <a href="../../../_sources/rados/troubleshooting/troubleshooting-mon.rst.txt" rel="nofollow"> View page source</a>
          
        
      </li>
    
  </ul>

  
  <hr/>
</div>
  </header>
  <div class="wy-grid-for-nav">
    
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search"  style="background: #eee" >
          

          
            <a href="../../../">
          

          
            
            <img src="../../../_static/logo.png" class="logo" alt="Logo"/>
          
          </a>

          

          
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../../../search/" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
  </form>
</div>

          
        </div>

        
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
          
            
            
              
            
            
              <ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../../../start/intro/">Ceph 简介</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../install/">安装 Ceph</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephadm/">Cephadm</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="../../">Ceph 存储集群</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="../../configuration/">配置</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../operations/">运维</a></li>
<li class="toctree-l2"><a class="reference internal" href="../../man/">    手册页</a></li>
<li class="toctree-l2 current"><a class="reference internal" href="../">故障排除</a><ul class="current">
<li class="toctree-l3"><a class="reference internal" href="../community/">Ceph 社区</a></li>
<li class="toctree-l3"><a class="reference internal" href="../log-and-debug/">日志记录和调试</a></li>
<li class="toctree-l3 current"><a class="current reference internal" href="#">监视器故障排除</a><ul>
<li class="toctree-l4"><a class="reference internal" href="#id2">开始排障</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id3">使用监视器的管理套接字</a></li>
<li class="toctree-l4"><a class="reference internal" href="#mon-status">理解 mon_status</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id4">最常见的监视器问题</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id8">监视器存储故障</a></li>
<li class="toctree-l4"><a class="reference internal" href="#id12">所有尝试都失败了，怎么办？</a></li>
</ul>
</li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting-osd/">OSD 故障排除</a></li>
<li class="toctree-l3"><a class="reference internal" href="../troubleshooting-pg/">归置组排障</a></li>
<li class="toctree-l3"><a class="reference internal" href="../memory-profiling/">内存剖析</a></li>
<li class="toctree-l3"><a class="reference internal" href="../cpu-profiling/">CPU 剖析</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="../../api/">APIs</a></li>
</ul>
</li>
<li class="toctree-l1"><a class="reference internal" href="../../../cephfs/">Ceph 文件系统</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../rbd/">Ceph 块设备</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../radosgw/">Ceph 对象网关</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/">Ceph 管理器守护进程</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../mgr/dashboard/">Ceph 仪表盘</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../api/">API 文档</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../architecture/">体系结构</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/developer_guide/">开发者指南</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../dev/internals/">Ceph 内幕</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../governance/">项目管理</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../foundation/">Ceph 基金会</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../ceph-volume/">ceph-volume</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/general/">Ceph 版本（总目录）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../releases/">Ceph 版本（索引）</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../security/">Security</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../glossary/">Ceph 术语</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../jaegertracing/">Tracing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../../../translation_cn/">中文版翻译资源</a></li>
</ul>

            
          
        </div>
        
      </div>
    </nav>

    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">

      
      <nav class="wy-nav-top" aria-label="top navigation">
        
          <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
          <a href="../../../">Ceph</a>
        
      </nav>


      <div class="wy-nav-content">
        
        <div class="rst-content">
        
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
            
<div id="dev-warning" class="admonition note">
  <p class="first admonition-title">Notice</p>
  <p class="last">This document is for a development version of Ceph.</p>
</div>
  <div id="docubetter" align="right" style="padding: 5px; font-weight: bold;">
    <a href="https://pad.ceph.com/p/Report_Documentation_Bugs">Report a Documentation Bug</a>
  </div>

  
  <div class="section" id="rados-troubleshooting-mon">
<span id="id1"></span><h1>监视器故障排除<a class="headerlink" href="#rados-troubleshooting-mon" title="Permalink to this headline">¶</a></h1>
<p id="index-0">When a cluster encounters monitor-related troubles there’s a tendency to
panic, and sometimes with good reason. Losing one or more monitors doesn’t
necessarily mean that your cluster is down, so long as a majority are up,
running, and form a quorum.
Regardless of how bad the situation is, the first thing you should do is to
calm down, take a breath, and step through the below troubleshooting steps.</p>
<div class="section" id="id2">
<h2>开始排障<a class="headerlink" href="#id2" title="Permalink to this headline">¶</a></h2>
<p><strong>监视器在运行吗？</strong></p>
<blockquote>
<div><p>First of all, we need to make sure the monitor (<em>mon</em>) daemon processes
(<code class="docutils literal notranslate"><span class="pre">ceph-mon</span></code>) are running.  You would be amazed by how often Ceph admins
forget to start the mons, or to restart them after an upgrade. There’s no
shame, but try to not lose a couple of hours looking for a deeper problem.
When running Kraken or later releases also ensure that the manager
daemons (<code class="docutils literal notranslate"><span class="pre">ceph-mgr</span></code>) are running, usually alongside each <code class="docutils literal notranslate"><span class="pre">ceph-mon</span></code>.</p>
</div></blockquote>
<p><strong>到各个监视器节点是否可达？</strong></p>
<blockquote>
<div><p>Doesn’t happen often, but sometimes there are <code class="docutils literal notranslate"><span class="pre">iptables</span></code> rules that
block accesse to mon nodes or TCP ports. These may be leftovers from
prior stress-testing or rule development. Try SSHing into
the server and, if that succeeds, try connecting to the monitor’s ports
(<code class="docutils literal notranslate"><span class="pre">tcp/3300</span></code> and <code class="docutils literal notranslate"><span class="pre">tcp/6789</span></code>) using a <code class="docutils literal notranslate"><span class="pre">telnet</span></code>, <code class="docutils literal notranslate"><span class="pre">nc</span></code>, or similar tools.</p>
</div></blockquote>
<p><strong>ceph -s 是否能运行并收到集群回复？</strong></p>
<blockquote>
<div><p>如果答案是肯定的，那么你的集群已启动并运行着。可以向你保证的是，监视器们必须形成法定人数之后才会响应 <code class="docutils literal notranslate"><span class="pre">status</span></code> 请求。
还必须确定能联系到至少一个 <code class="docutils literal notranslate"><span class="pre">mgr</span></code> 守护进程，最好是全部都在。</p>
<p>If <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">-s</span></code> hangs without obtaining a reply from the cluster
or showing <code class="docutils literal notranslate"><span class="pre">fault</span></code> messages, then it is likely that your monitors
are either down completely or just a fraction are up – a fraction
insufficient to form a majority quorum.  This check will connect to an
arbitrary mon; in rare cases it may be illuminating to bind to specific
mons in sequence by adding e.g. <code class="docutils literal notranslate"><span class="pre">-m</span> <span class="pre">mymon1</span></code> to the command.</p>
</div></blockquote>
<p><strong>ceph -s 没返回是什么情况？</strong></p>
<blockquote>
<div><p>如果你还没跑通所有步骤，先回头做完。</p>
<p>你可以单独连线各监视器去询问其状态，不论是否达成法定人数。用 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">tell</span> <span class="pre">mon.ID</span> <span class="pre">mon_status</span></code> 命令，其中 ID 是监视器的标识符，它是针对集群内单个监视器的命令。在<a class="reference internal" href="#mon-status">理解 mon_status</a> 小节中，我们将会解读此命令的输出。</p>
<p>你还可以 ssh 登录到各监视器节点、并查看其守护进程的
admin socket （管理套接字）。</p>
</div></blockquote>
</div>
<div class="section" id="id3">
<h2>使用监视器的管理套接字<a class="headerlink" href="#id3" title="Permalink to this headline">¶</a></h2>
<p>通过管理套接字，你可以用 Unix 套接字文件直接与指定守护进程交互。这个文件位于你监视器的 <code class="docutils literal notranslate"><span class="pre">run</span></code> 目录下，默认配置时它位于 <code class="docutils literal notranslate"><span class="pre">/var/run/ceph/ceph-mon.ID.asok</span></code> ，但你要是改过就不一定在那里了。如果你在那里没找到它，请看看 <code class="docutils literal notranslate"><span class="pre">ceph.conf</span></code> 里是否配置了其它路径、或者用下面的命令获取：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span><span class="o">-</span><span class="n">conf</span> <span class="o">--</span><span class="n">name</span> <span class="n">mon</span><span class="o">.</span><span class="n">ID</span> <span class="o">--</span><span class="n">show</span><span class="o">-</span><span class="n">config</span><span class="o">-</span><span class="n">value</span> <span class="n">admin_socket</span>
</pre></div>
</div>
<p>请牢记，只有在监视器运行时管理套接字才可用。监视器正常关闭时，管理套接字会被删除；如果监视器不运行了、但管理套接字还存在，就说明监视器不是正常关闭的。不管怎样，监视器没在运行，你就不能使用管理套接字， <code class="docutils literal notranslate"><span class="pre">ceph</span></code> 命令会返回类似 <code class="docutils literal notranslate"><span class="pre">Error</span> <span class="pre">111:</span> <span class="pre">Connection</span> <span class="pre">Refused</span></code> 的错误消息。</p>
<p>Accessing the admin socket is as simple as running <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">tell</span></code> on the daemon
you are interested in. For example:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">tell</span> <span class="n">mon</span><span class="o">.&lt;</span><span class="nb">id</span><span class="o">&gt;</span> <span class="n">mon_status</span>
</pre></div>
</div>
<p>Under the hood, this passes the command <code class="docutils literal notranslate"><span class="pre">help</span></code> to the running MON daemon
<code class="docutils literal notranslate"><span class="pre">&lt;id&gt;</span></code> via its “admin socket”, which is a file ending in <code class="docutils literal notranslate"><span class="pre">.asok</span></code>
somewhere under <code class="docutils literal notranslate"><span class="pre">/var/run/ceph</span></code>. Once you know the full path to the file,
you can even do this yourself:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="o">--</span><span class="n">admin</span><span class="o">-</span><span class="n">daemon</span> <span class="o">&lt;</span><span class="n">full_path_to_asok_file</span><span class="o">&gt;</span> <span class="o">&lt;</span><span class="n">command</span><span class="o">&gt;</span>
</pre></div>
</div>
<p><code class="docutils literal notranslate"><span class="pre">ceph</span></code> 工具的 <code class="docutils literal notranslate"><span class="pre">help</span></code> 命令会显示管理套接字支持的其它命令。请仔细了解一下 <code class="docutils literal notranslate"><span class="pre">config</span> <span class="pre">get</span></code> 、 <code class="docutils literal notranslate"><span class="pre">config</span> <span class="pre">show</span></code> 、 <code class="docutils literal notranslate"><span class="pre">mon_status</span></code> 和 <code class="docutils literal notranslate"><span class="pre">quorum_status</span></code> 命令，在排除监视器故障时它们能给你些启发。</p>
</div>
<div class="section" id="mon-status">
<h2>理解 mon_status<a class="headerlink" href="#mon-status" title="Permalink to this headline">¶</a></h2>
<p><code class="docutils literal notranslate"><span class="pre">mon_status</span></code> can always be obtained via the admin socket. This command will
output a multitude of information about the monitor, including the same output
you would get with <code class="docutils literal notranslate"><span class="pre">quorum_status</span></code>.</p>
<p>Take the following example output of <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">tell</span> <span class="pre">mon.c</span> <span class="pre">mon_status</span></code>:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="p">{</span> <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;c&quot;</span><span class="p">,</span>
  <span class="s2">&quot;rank&quot;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
  <span class="s2">&quot;state&quot;</span><span class="p">:</span> <span class="s2">&quot;peon&quot;</span><span class="p">,</span>
  <span class="s2">&quot;election_epoch&quot;</span><span class="p">:</span> <span class="mi">38</span><span class="p">,</span>
  <span class="s2">&quot;quorum&quot;</span><span class="p">:</span> <span class="p">[</span>
        <span class="mi">1</span><span class="p">,</span>
        <span class="mi">2</span><span class="p">],</span>
  <span class="s2">&quot;outside_quorum&quot;</span><span class="p">:</span> <span class="p">[],</span>
  <span class="s2">&quot;extra_probe_peers&quot;</span><span class="p">:</span> <span class="p">[],</span>
  <span class="s2">&quot;sync_provider&quot;</span><span class="p">:</span> <span class="p">[],</span>
  <span class="s2">&quot;monmap&quot;</span><span class="p">:</span> <span class="p">{</span> <span class="s2">&quot;epoch&quot;</span><span class="p">:</span> <span class="mi">3</span><span class="p">,</span>
      <span class="s2">&quot;fsid&quot;</span><span class="p">:</span> <span class="s2">&quot;5c4e9d53-e2e1-478a-8061-f543f8be4cf8&quot;</span><span class="p">,</span>
      <span class="s2">&quot;modified&quot;</span><span class="p">:</span> <span class="s2">&quot;2013-10-30 04:12:01.945629&quot;</span><span class="p">,</span>
      <span class="s2">&quot;created&quot;</span><span class="p">:</span> <span class="s2">&quot;2013-10-29 14:14:41.914786&quot;</span><span class="p">,</span>
      <span class="s2">&quot;mons&quot;</span><span class="p">:</span> <span class="p">[</span>
            <span class="p">{</span> <span class="s2">&quot;rank&quot;</span><span class="p">:</span> <span class="mi">0</span><span class="p">,</span>
              <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;a&quot;</span><span class="p">,</span>
              <span class="s2">&quot;addr&quot;</span><span class="p">:</span> <span class="s2">&quot;127.0.0.1:6789\/0&quot;</span><span class="p">},</span>
            <span class="p">{</span> <span class="s2">&quot;rank&quot;</span><span class="p">:</span> <span class="mi">1</span><span class="p">,</span>
              <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;b&quot;</span><span class="p">,</span>
              <span class="s2">&quot;addr&quot;</span><span class="p">:</span> <span class="s2">&quot;127.0.0.1:6790\/0&quot;</span><span class="p">},</span>
            <span class="p">{</span> <span class="s2">&quot;rank&quot;</span><span class="p">:</span> <span class="mi">2</span><span class="p">,</span>
              <span class="s2">&quot;name&quot;</span><span class="p">:</span> <span class="s2">&quot;c&quot;</span><span class="p">,</span>
              <span class="s2">&quot;addr&quot;</span><span class="p">:</span> <span class="s2">&quot;127.0.0.1:6795\/0&quot;</span><span class="p">}]}}</span>
</pre></div>
</div>
<p>A couple of things are obvious: we have three monitors in the monmap (<em>a</em>, <em>b</em>
and <em>c</em>), the quorum is formed by only two monitors, and <em>c</em> is in the quorum
as a <em>peon</em>.</p>
<p>Which monitor is out of the quorum?</p>
<blockquote>
<div><p>The answer would be <strong>a</strong>.</p>
</div></blockquote>
<p>Why?</p>
<blockquote>
<div><p>Take a look at the <code class="docutils literal notranslate"><span class="pre">quorum</span></code> set. We have two monitors in this set: <em>1</em>
and <em>2</em>. These are not monitor names. These are monitor ranks, as established
in the current monmap. We are missing the monitor with rank 0, and according
to the monmap that would be <code class="docutils literal notranslate"><span class="pre">mon.a</span></code>.</p>
</div></blockquote>
<p>By the way, how are ranks established?</p>
<blockquote>
<div><p>Ranks are (re)calculated whenever you add or remove monitors and follow a
simple rule: the <strong>greater</strong> the <code class="docutils literal notranslate"><span class="pre">IP:PORT</span></code> combination, the <strong>lower</strong> the
rank is. In this case, considering that <code class="docutils literal notranslate"><span class="pre">127.0.0.1:6789</span></code> is lower than all
the remaining <code class="docutils literal notranslate"><span class="pre">IP:PORT</span></code> combinations, <code class="docutils literal notranslate"><span class="pre">mon.a</span></code> has rank 0.</p>
</div></blockquote>
</div>
<div class="section" id="id4">
<h2>最常见的监视器问题<a class="headerlink" href="#id4" title="Permalink to this headline">¶</a></h2>
<div class="section" id="id5">
<h3>存在法定人数但是挂了不止一个监视器<a class="headerlink" href="#id5" title="Permalink to this headline">¶</a></h3>
<p>When this happens, depending on the version of Ceph you are running,
you should be seeing something similar to:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph health detail
[snip]
mon.a (rank 0) addr 127.0.0.1:6789/0 is down (out of quorum)
</pre></div>
</div>
<p>How to troubleshoot this?</p>
<blockquote>
<div><p>First, make sure <code class="docutils literal notranslate"><span class="pre">mon.a</span></code> is running.</p>
<p>Second, make sure you are able to connect to <code class="docutils literal notranslate"><span class="pre">mon.a</span></code>’s node from the
other mon nodes. Check the TCP ports as well. Check <code class="docutils literal notranslate"><span class="pre">iptables</span></code> and
<code class="docutils literal notranslate"><span class="pre">nf_conntrack</span></code> on all nodes and ensure that you are not
dropping/rejecting connections.</p>
<p>If this initial troubleshooting doesn’t solve your problems, then it’s
time to go deeper.</p>
<p>First, check the problematic monitor’s <code class="docutils literal notranslate"><span class="pre">mon_status</span></code> via the admin
socket as explained in <a class="reference internal" href="#id3">使用监视器的管理套接字</a> and
<a class="reference internal" href="#mon-status">理解 mon_status</a>.</p>
<p>If the monitor is out of the quorum, its state should be one of
<code class="docutils literal notranslate"><span class="pre">probing</span></code>, <code class="docutils literal notranslate"><span class="pre">electing</span></code> or <code class="docutils literal notranslate"><span class="pre">synchronizing</span></code>. If it happens to be either
<code class="docutils literal notranslate"><span class="pre">leader</span></code> or <code class="docutils literal notranslate"><span class="pre">peon</span></code>, then the monitor believes to be in quorum, while
the remaining cluster is sure it is not; or maybe it got into the quorum
while we were troubleshooting the monitor, so check you <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">-s</span></code> again
just to make sure. Proceed if the monitor is not yet in the quorum.</p>
</div></blockquote>
<p>What if the state is <code class="docutils literal notranslate"><span class="pre">probing</span></code>?</p>
<blockquote>
<div><p>This means the monitor is still looking for the other monitors. Every time
you start a monitor, the monitor will stay in this state for some time
while trying to find the rest of the monitors specified in the <code class="docutils literal notranslate"><span class="pre">monmap</span></code>.
The time a monitor will spend in this state can vary. For instance, when on
a single-monitor cluster, the monitor will pass through the probing state
almost instantaneously, since there are no other monitors around. On a
multi-monitor cluster, the monitors will stay in this state until they
find enough monitors to form a quorum – this means that if you have 2 out
of 3 monitors down, the one remaining monitor will stay in this state
indefinitively until you bring one of the other monitors up.</p>
<p>If you have a quorum, however, the monitor should be able to find the
remaining monitors pretty fast, as long as they can be reached. If your
monitor is stuck probing and you’ve gone through with all the communication
troubleshooting, then there is a fair chance that the monitor is trying
to reach the other monitors on a wrong address. <code class="docutils literal notranslate"><span class="pre">mon_status</span></code> outputs the
<code class="docutils literal notranslate"><span class="pre">monmap</span></code> known to the monitor: check if the other monitor’s locations
match reality. If they don’t, jump to
<a class="reference internal" href="#monmap">修复监视器损坏的 monmap</a>; if they do, then it may be related
to severe clock skews amongst the monitor nodes and you should refer to
<a class="reference internal" href="#id6">时钟偏移</a> first, but if that doesn’t solve your problem then it is
the time to prepare some logs and reach out to the community (please refer
to <a class="reference internal" href="#id14">收集所需日志</a> on how to best prepare your logs).</p>
</div></blockquote>
<p>What if state is <code class="docutils literal notranslate"><span class="pre">electing</span></code>?</p>
<blockquote>
<div><p>This means the monitor is in the middle of an election. With recent Ceph
releases these typically complete quickly, but at times the monitors can
get stuck in what is known as an <em>election storm</em>. This can indicate
clock skew among the monitor nodes; jump to
<a class="reference internal" href="#id6">时钟偏移</a> for more information. If all your clocks are properly
synchronized, you should search the mailing lists and tracker.
This is not a state that is likely to persist and aside from
(<em>really</em>) old bugs there is not an obvious reason besides clock skews on
why this would happen.  Worst case, if there are enough surviving mons,
down the problematic one while you investigate.</p>
</div></blockquote>
<p>What if state is <code class="docutils literal notranslate"><span class="pre">synchronizing</span></code>?</p>
<blockquote>
<div><p>This means the monitor is synchronizing with the rest of the cluster in
order to join the quorum. The synchronization process is as faster as
smaller your monitor store is, so if you have a big store it may
take a while. Don’t worry, it should be finished soon enough.</p>
<p>However, if you notice that the monitor jumps from <code class="docutils literal notranslate"><span class="pre">synchronizing</span></code> to
<code class="docutils literal notranslate"><span class="pre">electing</span></code> and then back to <code class="docutils literal notranslate"><span class="pre">synchronizing</span></code>, then you do have a
problem: the cluster state is advancing (i.e., generating new maps) way
too fast for the synchronization process to keep up. This used to be a
thing in early Cuttlefish, but since then the synchronization process was
quite refactored and enhanced to avoid just this sort of behavior. If this
happens in later versions let us know. And bring some logs
(see <a class="reference internal" href="#id14">收集所需日志</a>).</p>
</div></blockquote>
<p>What if state is <code class="docutils literal notranslate"><span class="pre">leader</span></code> or <code class="docutils literal notranslate"><span class="pre">peon</span></code>?</p>
<blockquote>
<div><p>This should not happen. There is a chance this might happen however, and
it has a lot to do with clock skews – see <a class="reference internal" href="#id6">时钟偏移</a>. If you’re not
suffering from clock skews, then please prepare your logs (see
<a class="reference internal" href="#id14">收集所需日志</a>) and reach out to us.</p>
</div></blockquote>
</div>
<div class="section" id="monmap">
<h3>修复监视器损坏的 monmap<a class="headerlink" href="#monmap" title="Permalink to this headline">¶</a></h3>
<p>This is how a <code class="docutils literal notranslate"><span class="pre">monmap</span></code> usually looks, depending on the number of
monitors:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">epoch</span> <span class="mi">3</span>
<span class="n">fsid</span> <span class="mi">5</span><span class="n">c4e9d53</span><span class="o">-</span><span class="n">e2e1</span><span class="o">-</span><span class="mi">478</span><span class="n">a</span><span class="o">-</span><span class="mi">8061</span><span class="o">-</span><span class="n">f543f8be4cf8</span>
<span class="n">last_changed</span> <span class="mi">2013</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">30</span> <span class="mi">04</span><span class="p">:</span><span class="mi">12</span><span class="p">:</span><span class="mf">01.945629</span>
<span class="n">created</span> <span class="mi">2013</span><span class="o">-</span><span class="mi">10</span><span class="o">-</span><span class="mi">29</span> <span class="mi">14</span><span class="p">:</span><span class="mi">14</span><span class="p">:</span><span class="mf">41.914786</span>
<span class="mi">0</span><span class="p">:</span> <span class="mf">127.0.0.1</span><span class="p">:</span><span class="mi">6789</span><span class="o">/</span><span class="mi">0</span> <span class="n">mon</span><span class="o">.</span><span class="n">a</span>
<span class="mi">1</span><span class="p">:</span> <span class="mf">127.0.0.1</span><span class="p">:</span><span class="mi">6790</span><span class="o">/</span><span class="mi">0</span> <span class="n">mon</span><span class="o">.</span><span class="n">b</span>
<span class="mi">2</span><span class="p">:</span> <span class="mf">127.0.0.1</span><span class="p">:</span><span class="mi">6795</span><span class="o">/</span><span class="mi">0</span> <span class="n">mon</span><span class="o">.</span><span class="n">c</span>
</pre></div>
</div>
<p>This may not be what you have however. For instance, in some versions of
early Cuttlefish there was a bug that could cause your <code class="docutils literal notranslate"><span class="pre">monmap</span></code>
to be nullified.  Completely filled with zeros. This means that not even
<code class="docutils literal notranslate"><span class="pre">monmaptool</span></code> would be able to make sense of cold, hard, inscrutable zeros.
It’s also possible to end up with a monitor with a severely outdated monmap,
notably if the node has been down for months while you fight with your vendor’s
TAC.  The subject <code class="docutils literal notranslate"><span class="pre">ceph-mon</span></code> daemon might be unable to find the surviving
monitors (e.g., say <code class="docutils literal notranslate"><span class="pre">mon.c</span></code> is down; you add a new monitor <code class="docutils literal notranslate"><span class="pre">mon.d</span></code>,
then remove <code class="docutils literal notranslate"><span class="pre">mon.a</span></code>, then add a new monitor <code class="docutils literal notranslate"><span class="pre">mon.e</span></code> and remove
<code class="docutils literal notranslate"><span class="pre">mon.b</span></code>; you will end up with a totally different monmap from the one
<code class="docutils literal notranslate"><span class="pre">mon.c</span></code> knows).</p>
<p>In this situation you have two possible solutions:</p>
<p>Scrap the monitor and redeploy</p>
<blockquote>
<div><p>You should only take this route if you are positive that you won’t
lose the information kept by that monitor; that you have other monitors
and that they are running just fine so that your new monitor is able
to synchronize from the remaining monitors. Keep in mind that destroying
a monitor, if there are no other copies of its contents, may lead to
loss of data.</p>
</div></blockquote>
<p>Inject a monmap into the monitor</p>
<blockquote>
<div><p>Usually the safest path. You should grab the monmap from the remaining
monitors and inject it into the monitor with the corrupted/lost monmap.</p>
<p>These are the basic steps:</p>
<ol class="arabic">
<li><p>Is there a formed quorum? If so, grab the monmap from the quorum:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph mon getmap -o /tmp/monmap
</pre></div>
</div>
</li>
<li><p>No quorum? Grab the monmap directly from another monitor (this
assumes the monitor you’re grabbing the monmap from has id ID-FOO
and has been stopped):</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph-mon -i ID-FOO --extract-monmap /tmp/monmap
</pre></div>
</div>
</li>
<li><p>Stop the monitor you’re going to inject the monmap into.</p></li>
<li><p>Inject the monmap:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span>$ ceph-mon -i ID --inject-monmap /tmp/monmap
</pre></div>
</div>
</li>
<li><p>Start the monitor</p></li>
</ol>
<p>Please keep in mind that the ability to inject monmaps is a powerful
feature that can cause havoc with your monitors if misused as it will
overwrite the latest, existing monmap kept by the monitor.</p>
</div></blockquote>
</div>
<div class="section" id="id6">
<h3>时钟偏移<a class="headerlink" href="#id6" title="Permalink to this headline">¶</a></h3>
<p>Monitor operation can be severely affected by clock skew among the quorum’s
mons, as the PAXOS consensus algorithm requires tight time alignment.
Skew can result in weird behavior with no obvious
cause. To avoid such issues, you must run a clock synchronization tool
on your monitor nodes:  <code class="docutils literal notranslate"><span class="pre">Chrony</span></code> or the legacy <code class="docutils literal notranslate"><span class="pre">ntpd</span></code>.  Be sure to
configure the mon nodes with the <cite>iburst</cite> option and multiple peers:</p>
<ul class="simple">
<li><p>Each other</p></li>
<li><p>Internal <code class="docutils literal notranslate"><span class="pre">NTP</span></code> servers</p></li>
<li><p>Multiple external, public pool servers</p></li>
</ul>
<p>For good measure, <em>all</em> nodes in your cluster should also sync against
internal and external servers, and perhaps even your mons.  <code class="docutils literal notranslate"><span class="pre">NTP</span></code> servers
should run on bare metal; VM virtualized clocks are not suitable for steady
timekeeping.  Visit <a class="reference external" href="https://www.ntp.org">https://www.ntp.org</a> for more info.  Your
organization may already have quality internal <code class="docutils literal notranslate"><span class="pre">NTP</span></code> servers you can use.
Sources for <code class="docutils literal notranslate"><span class="pre">NTP</span></code> server appliances include:</p>
<ul class="simple">
<li><p>Microsemi (formerly Symmetricom) <a class="reference external" href="https://www.microsemi.com/product-directory/3425-timing-synchronization">https://microsemi.com</a></p></li>
<li><p>EndRun <a class="reference external" href="https://endruntechnologies.com/products/ntp-time-servers">https://endruntechnologies.com</a></p></li>
<li><p>Netburner <a class="reference external" href="https://www.netburner.com/products/network-time-server/pk70-ex-ntp-network-time-server">https://www.netburner.com</a></p></li>
</ul>
<p>What’s the maximum tolerated clock skew?</p>
<blockquote>
<div><p>By default the monitors will allow clocks to drift up to 0.05 seconds (50 ms).</p>
</div></blockquote>
<p>Can I increase the maximum tolerated clock skew?</p>
<blockquote>
<div><p>The maximum tolerated clock skew is configurable via the
<code class="docutils literal notranslate"><span class="pre">mon-clock-drift-allowed</span></code> option, and
although you <em>CAN</em> you almost certainly <em>SHOULDN’T</em>. The clock skew mechanism
is in place because clock-skewed monitors are liely to misbehave. We, as
developers and QA aficionados, are comfortable with the current default
value, as it will alert the user before the monitors get out hand. Changing
this value may cause unforeseen effects on the
stability of the monitors and overall cluster health.</p>
</div></blockquote>
<p>How do I know there’s a clock skew?</p>
<blockquote>
<div><p>The monitors will warn you via the cluster status <code class="docutils literal notranslate"><span class="pre">HEALTH_WARN</span></code>. <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">health</span>
<span class="pre">detail</span></code> or <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">status</span></code> should show something like:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">mon</span><span class="o">.</span><span class="n">c</span> <span class="n">addr</span> <span class="mf">10.10.0.1</span><span class="p">:</span><span class="mi">6789</span><span class="o">/</span><span class="mi">0</span> <span class="n">clock</span> <span class="n">skew</span> <span class="mf">0.08235</span><span class="n">s</span> <span class="o">&gt;</span> <span class="nb">max</span> <span class="mf">0.05</span><span class="n">s</span> <span class="p">(</span><span class="n">latency</span> <span class="mf">0.0045</span><span class="n">s</span><span class="p">)</span>
</pre></div>
</div>
<p>That means that <code class="docutils literal notranslate"><span class="pre">mon.c</span></code> has been flagged as suffering from a clock skew.</p>
<p>On releases beginning with Luminous you can issue the
<code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">time-sync-status</span></code> command to check status.  Note that the lead mon
is typically the one with the numerically lowest IP address.  It will always
show <code class="docutils literal notranslate"><span class="pre">0</span></code>: the reported offsets of other mons are relative to
the lead mon, not to any external reference source.</p>
</div></blockquote>
<p>What should I do if there’s a clock skew?</p>
<blockquote>
<div><p>Synchronize your clocks. Running an NTP client may help. If you are already
using one and you hit this sort of issues, check if you are using some NTP
server remote to your network and consider hosting your own NTP server on
your network.  This last option tends to reduce the amount of issues with
monitor clock skews.</p>
</div></blockquote>
</div>
<div class="section" id="id7">
<h3>客户端不能连接或挂载<a class="headerlink" href="#id7" title="Permalink to this headline">¶</a></h3>
<p>检查防火墙配置。有些系统安装工具把 <code class="docutils literal notranslate"><span class="pre">REJECT</span></code> 规则加入了
<code class="docutils literal notranslate"><span class="pre">iptables</span></code> ，它会拒绝除 <code class="docutils literal notranslate"><span class="pre">ssh</span></code> 以外的所有入栈连接。如果你的监视器主机有这样的 <code class="docutils literal notranslate"><span class="pre">REJECT</span></code> 规则，别的客户端进来的连接将遇到超时错误而不能挂载。得先找到这条拒绝客户端连入的 <code class="docutils literal notranslate"><span class="pre">iptables</span></code> 规则，例如，你要找到形似以下的规则：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">REJECT</span> <span class="nb">all</span> <span class="o">--</span> <span class="n">anywhere</span> <span class="n">anywhere</span> <span class="n">reject</span><span class="o">-</span><span class="k">with</span> <span class="n">icmp</span><span class="o">-</span><span class="n">host</span><span class="o">-</span><span class="n">prohibited</span>
</pre></div>
</div>
<p>你也许还要在 Ceph 主机上增加 iptables 规则来放通 Ceph 监视器端口（即默认的 6789 端口）、和 OSD 端口（默认从 6800 到 7300
）。例如：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">iptables</span> <span class="o">-</span><span class="n">A</span> <span class="n">INPUT</span> <span class="o">-</span><span class="n">m</span> <span class="n">multiport</span> <span class="o">-</span><span class="n">p</span> <span class="n">tcp</span> <span class="o">-</span><span class="n">s</span> <span class="p">{</span><span class="n">ip</span><span class="o">-</span><span class="n">address</span><span class="p">}</span><span class="o">/</span><span class="p">{</span><span class="n">netmask</span><span class="p">}</span> <span class="o">--</span><span class="n">dports</span> <span class="mi">6789</span><span class="p">,</span><span class="mi">6800</span><span class="p">:</span><span class="mi">7300</span> <span class="o">-</span><span class="n">j</span> <span class="n">ACCEPT</span>
</pre></div>
</div>
</div>
</div>
<div class="section" id="id8">
<h2>监视器存储故障<a class="headerlink" href="#id8" title="Permalink to this headline">¶</a></h2>
<div class="section" id="id9">
<h3>存储损坏的症状<a class="headerlink" href="#id9" title="Permalink to this headline">¶</a></h3>
<p>Ceph 监视器把<a class="reference internal" href="../../../glossary/#term-12"><span class="xref std std-term">集群运行图</span></a>存储在键值数据库里，像
LevelDB 。如果某个监视器由于键值存储损坏而失败，监视器日志里可能出现如下错误消息：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Corruption</span><span class="p">:</span> <span class="n">error</span> <span class="ow">in</span> <span class="n">middle</span> <span class="n">of</span> <span class="n">record</span>
</pre></div>
</div>
<p>或者：</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">Corruption</span><span class="p">:</span> <span class="mi">1</span> <span class="n">missing</span> <span class="n">files</span><span class="p">;</span> <span class="n">e</span><span class="o">.</span><span class="n">g</span><span class="o">.</span><span class="p">:</span> <span class="o">/</span><span class="n">var</span><span class="o">/</span><span class="n">lib</span><span class="o">/</span><span class="n">ceph</span><span class="o">/</span><span class="n">mon</span><span class="o">/</span><span class="n">mon</span><span class="o">.</span><span class="n">foo</span><span class="o">/</span><span class="n">store</span><span class="o">.</span><span class="n">db</span><span class="o">/</span><span class="mf">1234567.</span><span class="n">ldb</span>
</pre></div>
</div>
</div>
<div class="section" id="id10">
<h3>用健康的监视器恢复<a class="headerlink" href="#id10" title="Permalink to this headline">¶</a></h3>
<p>只要有幸存的，我们就可以用新的<a class="reference internal" href="../../operations/add-or-rm-mons/#adding-and-removing-monitors"><span class="std std-ref">替换掉</span></a>损坏的；而且新加入的监视器启动后会与健康节点同步，完全同步后就可以服务于客户端了。</p>
</div>
<div class="section" id="osd">
<span id="mon-store-recovery-using-osds"></span><h3>用 OSD 恢复<a class="headerlink" href="#osd" title="Permalink to this headline">¶</a></h3>
<p>但是，所有监视器同时失效怎么办呢？我们建议用户在一个 Ceph 集群内至少部署三个监视器（最好是五个），所以同时失效的可能性非常低。但是，计划外的数据中心掉电、加上配置不当的磁盘和文件系统可能致使底层文件系统损坏，并因此损坏所有监视器。在这种情况下，我们可以用存储在 OSD 上的信息恢复监视器存储。</p>
<div class="highlight-bash notranslate"><div class="highlight"><pre><span></span><span class="nv">ms</span><span class="o">=</span>/root/mon-store
mkdir <span class="nv">$ms</span>

<span class="c1"># 从已关停的 OSD 收集集群运行图</span>
<span class="k">for</span> host <span class="k">in</span> <span class="nv">$hosts</span><span class="p">;</span> <span class="k">do</span>
  rsync -avz <span class="nv">$ms</span>/. user@<span class="nv">$host</span>:<span class="nv">$ms</span>.remote
  rm -rf <span class="nv">$ms</span>
  ssh user@<span class="nv">$host</span> <span class="s">&lt;&lt;EOF</span>
<span class="s">    for osd in /var/lib/ceph/osd/ceph-*; do</span>
<span class="s">      ceph-objectstore-tool --data-path \$osd --no-mon-config --op update-mon-db --mon-store-path $ms.remote</span>
<span class="s">    done</span>
<span class="s">EOF</span>
  rsync -avz user@<span class="nv">$host</span>:<span class="nv">$ms</span>.remote/. <span class="nv">$ms</span>
<span class="k">done</span>

<span class="c1"># 用收集来的运行图重建监视器存储，如果集群没用 cephx 认证，\</span>
<span class="c1"># 我们可以跳过更新密钥环的步骤，也不用加 --keyring 选项了，\</span>
<span class="c1"># 就是说可以直接运行 ``ceph-monstore-tool $ms rebuild``</span>
ceph-authtool /path/to/admin.keyring -n mon. <span class="se">\</span>
  --cap mon <span class="s1">&#39;allow *&#39;</span>
ceph-authtool /path/to/admin.keyring -n client.admin <span class="se">\</span>
  --cap mon <span class="s1">&#39;allow *&#39;</span> --cap osd <span class="s1">&#39;allow *&#39;</span> --cap mds <span class="s1">&#39;allow *&#39;</span>
<span class="c1"># add one or more ceph-mgr&#39;s key to the keyring. in this case, an encoded key</span>
<span class="c1"># for mgr.x is added, you can find the encoded key in</span>
<span class="c1"># /etc/ceph/${cluster}.${mgr_name}.keyring on the machine where ceph-mgr is</span>
<span class="c1"># deployed</span>
ceph-authtool /path/to/admin.keyring --add-key <span class="s1">&#39;AQDN8kBe9PLWARAAZwxXMr+n85SBYbSlLcZnMA==&#39;</span> -n mgr.x <span class="se">\</span>
  --cap mon <span class="s1">&#39;allow profile mgr&#39;</span> --cap osd <span class="s1">&#39;allow *&#39;</span> --cap mds <span class="s1">&#39;allow *&#39;</span>
<span class="c1"># If your monitors&#39; ids are not sorted by ip address, please specify them in order.</span>
<span class="c1"># For example. if mon &#39;a&#39; is 10.0.0.3, mon &#39;b&#39; is 10.0.0.2, and mon &#39;c&#39; is  10.0.0.4,</span>
<span class="c1"># please passing &quot;--mon-ids b a c&quot;.</span>
<span class="c1"># In addition, if your monitors&#39; ids are not single characters like &#39;a&#39;, &#39;b&#39;, &#39;c&#39;, please</span>
<span class="c1"># specify them in the command line by passing them as arguments of the &quot;--mon-ids&quot;</span>
<span class="c1"># option. if you are not sure, please check your ceph.conf to see if there is any</span>
<span class="c1"># sections named like &#39;[mon.foo]&#39;. don&#39;t pass the &quot;--mon-ids&quot; option, if you are</span>
<span class="c1"># using DNS SRV for looking up monitors.</span>
ceph-monstore-tool <span class="nv">$ms</span> rebuild -- --keyring /path/to/admin.keyring --mon-ids alpha beta gamma

<span class="c1"># 备份一下损坏的 store.db 以防万一！所有监视器上都要备份一下。</span>
mv /var/lib/ceph/mon/mon.foo/store.db /var/lib/ceph/mon/mon.foo/store.db.corrupted

<span class="c1"># move rebuild store.db into place.  repeat for all monitors.</span>
mv <span class="nv">$ms</span>/store.db /var/lib/ceph/mon/mon.foo/store.db
chown -R ceph:ceph /var/lib/ceph/mon/mon.foo/store.db
</pre></div>
</div>
<p>上面的步骤</p>
<ol class="arabic simple">
<li><p>从所有 OSD 收集映射图</p></li>
<li><p>然后重建监视器存储</p></li>
<li><p>把各项目加进密钥环文件，并分配相应的能力</p></li>
<li><p>用恢复好的副本替换 <code class="docutils literal notranslate"><span class="pre">mon.foo</span></code> 上损坏的存储。</p></li>
</ol>
<div class="section" id="id11">
<h4>已知的局限性<a class="headerlink" href="#id11" title="Permalink to this headline">¶</a></h4>
<p>通过上面的步骤无法恢复以下信息：</p>
<ul class="simple">
<li><p><strong>一些加过的密钥环</strong>：所有用 <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">auth</span> <span class="pre">add</span></code> 命令加上的 OSD 密钥环都从 OSD 副本中恢复了； <code class="docutils literal notranslate"><span class="pre">client.admin</span></code> 密钥环也用
<code class="docutils literal notranslate"><span class="pre">ceph-monstore-tool</span></code> 导入了。但是 MDS 密钥环和其它密钥环却丢失了，你也许得手动重加。</p></li>
<li><p><strong>正在创建的存储池</strong>: If any RADOS pools were in the process of being creating, that state is lost.  The recovery tool assumes that all pools have been created.  If there are PGs that are stuck in the ‘unknown’ after the recovery for a partially created pool, you can force creation of the <em>empty</em> PG with the <code class="docutils literal notranslate"><span class="pre">ceph</span> <span class="pre">osd</span> <span class="pre">force-create-pg</span></code> command.  Note that this will create an <em>empty</em> PG, so only do this if you know the pool is empty.</p></li>
<li><p><strong>MDS 映射图</strong>： MDS 的各种映射图会丢失。</p></li>
</ul>
</div>
</div>
</div>
<div class="section" id="id12">
<h2>所有尝试都失败了，怎么办？<a class="headerlink" href="#id12" title="Permalink to this headline">¶</a></h2>
<div class="section" id="id13">
<h3>到外面寻求帮助<a class="headerlink" href="#id13" title="Permalink to this headline">¶</a></h3>
<p>You can find us on IRC at #ceph and #ceph-devel at OFTC (server irc.oftc.net)
and on <code class="docutils literal notranslate"><span class="pre">dev&#64;ceph.io</span></code> and <code class="docutils literal notranslate"><span class="pre">ceph-users&#64;lists.ceph.com</span></code>. Make
sure you have grabbed your logs and have them ready if someone asks: the faster
the interaction and lower the latency in response, the better chances everyone’s
time is optimized.</p>
</div>
<div class="section" id="id14">
<h3>收集所需日志<a class="headerlink" href="#id14" title="Permalink to this headline">¶</a></h3>
<p>Monitor logs are, by default, kept in <code class="docutils literal notranslate"><span class="pre">/var/log/ceph/ceph-mon.FOO.log*</span></code>. We
may want them. However, your logs may not have the necessary information. If
you don’t find your monitor logs at their default location, you can check
where they should be by running:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span><span class="o">-</span><span class="n">conf</span> <span class="o">--</span><span class="n">name</span> <span class="n">mon</span><span class="o">.</span><span class="n">FOO</span> <span class="o">--</span><span class="n">show</span><span class="o">-</span><span class="n">config</span><span class="o">-</span><span class="n">value</span> <span class="n">log_file</span>
</pre></div>
</div>
<p>The amount of information in the logs are subject to the debug levels being
enforced by your configuration files. If you have not enforced a specific
debug level then Ceph is using the default levels and your logs may not
contain important information to track down you issue.
A first step in getting relevant information into your logs will be to raise
debug levels. In this case we will be interested in the information from the
monitor.
Similarly to what happens on other components, different parts of the monitor
will output their debug information on different subsystems.</p>
<p>You will have to raise the debug levels of those subsystems more closely
related to your issue. This may not be an easy task for someone unfamiliar
with troubleshooting Ceph. For most situations, setting the following options
on your monitors will be enough to pinpoint a potential source of the issue:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">debug</span> <span class="n">mon</span> <span class="o">=</span> <span class="mi">10</span>
<span class="n">debug</span> <span class="n">ms</span> <span class="o">=</span> <span class="mi">1</span>
</pre></div>
</div>
<p>If we find that these debug levels are not enough, there’s a chance we may
ask you to raise them or even define other debug subsystems to obtain infos
from – but at least we started off with some useful information, instead
of a massively empty log without much to go on with.</p>
</div>
<div class="section" id="id15">
<h3>我需要重启监视器来更改调试级别吗？<a class="headerlink" href="#id15" title="Permalink to this headline">¶</a></h3>
<p>No. You may do it in one of two ways:</p>
<p>You have quorum</p>
<blockquote>
<div><p>Either inject the debug option into the monitor you want to debug:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">tell</span> <span class="n">mon</span><span class="o">.</span><span class="n">FOO</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">debug_mon</span> <span class="mi">10</span><span class="o">/</span><span class="mi">10</span>
</pre></div>
</div>
<p>or into all monitors at once:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">tell</span> <span class="n">mon</span><span class="o">.*</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">debug_mon</span> <span class="mi">10</span><span class="o">/</span><span class="mi">10</span>
</pre></div>
</div>
</div></blockquote>
<p>No quourm</p>
<blockquote>
<div><p>Use the monitor’s admin socket and directly adjust the configuration
options:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">daemon</span> <span class="n">mon</span><span class="o">.</span><span class="n">FOO</span> <span class="n">config</span> <span class="nb">set</span> <span class="n">debug_mon</span> <span class="mi">10</span><span class="o">/</span><span class="mi">10</span>
</pre></div>
</div>
</div></blockquote>
<p>Going back to default values is as easy as rerunning the above commands
using the debug level <code class="docutils literal notranslate"><span class="pre">1/10</span></code> instead.  You can check your current
values using the admin socket and the following commands:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">daemon</span> <span class="n">mon</span><span class="o">.</span><span class="n">FOO</span> <span class="n">config</span> <span class="n">show</span>
</pre></div>
</div>
<p>or:</p>
<div class="highlight-default notranslate"><div class="highlight"><pre><span></span><span class="n">ceph</span> <span class="n">daemon</span> <span class="n">mon</span><span class="o">.</span><span class="n">FOO</span> <span class="n">config</span> <span class="n">get</span> <span class="s1">&#39;OPTION_NAME&#39;</span>
</pre></div>
</div>
</div>
<div class="section" id="id16">
<h3>在某个调试级别下重现了问题，然后呢？<a class="headerlink" href="#id16" title="Permalink to this headline">¶</a></h3>
<p>Ideally you would send us only the relevant portions of your logs.
We realise that figuring out the corresponding portion may not be the
easiest of tasks. Therefore, we won’t hold it to you if you provide the
full log, but common sense should be employed. If your log has hundreds
of thousands of lines, it may get tricky to go through the whole thing,
specially if we are not aware at which point, whatever your issue is,
happened. For instance, when reproducing, keep in mind to write down
current time and date and to extract the relevant portions of your logs
based on that.</p>
<p>Finally, you should reach out to us on the mailing lists, on IRC or file
a new issue on the <a class="reference external" href="http://tracker.ceph.com/projects/ceph/issues/new">tracker</a>.</p>
</div>
</div>
</div>



           </div>
           
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="../troubleshooting-osd/" class="btn btn-neutral float-right" title="OSD 故障排除" accesskey="n" rel="next">Next <span class="fa fa-arrow-circle-right" aria-hidden="true"></span></a>
        <a href="../community/" class="btn btn-neutral float-left" title="Ceph 社区" accesskey="p" rel="prev"><span class="fa fa-arrow-circle-left" aria-hidden="true"></span> Previous</a>
    </div>

  <hr/>

  <div role="contentinfo">
    <p>
        &#169; Copyright 2016, Ceph authors and contributors. Licensed under Creative Commons Attribution Share Alike 3.0 (CC-BY-SA-3.0).

    </p>
  </div> 

</footer>
        </div>
      </div>

    </section>

  </div>
  

  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.Navigation.enable(true);
      });
  </script>

  
  
    
   

</body>
</html>